Dominoes and Storyboards: Beyond "Icons on Strings"

Practically since graphic displays were first hooked to computers, the idea of representing computer programs by pictures has attracted researchers. However, to date, most proposals for visual programming languages have adhered to a set pattern: fixed pictures symbolizing program components, connected by lines or arrows symbolizing relationships between the program components. This "icons on strings" approach, while it can be useful, is not the only way of visualizing programs. In this paper, I explore one alternative: representing a program through visual examples of the state of its execution. I present two related techniques: dominoes, which replace the traditional icons as representations of operations; and storyboards, which replace iconic circuitry as the representation of program code. These have been implemented in Mondrian, a graphic editor extensible through programming by example. Most proposed visual programming languages are "icons on strings" The dream of visual programming has been to replace textual programming languages as the medium through which computers are instructed with a language of pictures. As with natural languages, there is no one visual language that is privileged: many provide sufficient expressive power, but emphasize different kinds of relations. Despite the apparent variety of proposed visual languages, most fall into a category I call icons on strings. These languages are characterized by • Iconic pictures, usually static, that have a one-to-one mapping to the concepts that make up the program, contained on the screen in a box of bounded size, and • Connections between them, usually represented visually as lines or arrows. The connections represent relationships between the concepts. Languages may differ in what relations they choose to represent. The illustration on the top of the next page, a mosaic of several systems from the past two years' Visual Languages conferences, shows some of the many icon-onstring languages. The only major variant of the icon-on-string idea has been to replace the "strings" as a means of connecting the icons with the relation of containment. Icons can be contained inside one another to express relationships like the function-argument relation.This might be called "icons in icons". Containment relations are limited to trees, whereas lines can express full graphs. Boxer [Abelson and diSessa] and VennLisp [Lakin 86] are examples of containment tree languages. I used the idea of containment in three dimensions in [Lieberman 89]. What's wrong with "icons on strings"? Like program code in textual languages, graph representations of code visually expose only the representations of program code and relations between program modules. However, many tasks in programming require understanding not only the relationship between program components, but the relationship between program components and the data being operated upon. This is especially true in debugging. Debugging is a visualization task that requires discovering which parts of a program are responsible for the production of incorrect data, or which program components need to be added to modified to produce desired objects which do not appear in the incorrect program. For this, it is necessary to present visually not just the code, or just the data, but the relations between code and data. Algorithm animation is a valuable debugging tool, because it can dynamically represent the temporal evolution of data in a program. However, without representation of the code itself and indications of the connections between code and data, the task of associating undesired behavior with the characteristics of the algorithm that produced it is left to the unassisted thought processes of the programmer. Another problem with graph representations is that they scale up poorly. The typical symptom is that in highly interconnected graphs, the density of connections reduces the visual impression to a "plate of spaghetti". Abstraction mechanisms, such as zoom and pan, level of detail control, link consolidation, browsers, and other interactive techniques have as yet been insufficiently explored to alleviate this concern. Concrete visual examples illustrate abstract